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Problem Statement 

There has been growing recognition of the devolution of human resource development (HRD) responsibilities to line 
managers (e.g. Bevan & Hayday, 1994; Bevan et ah, 1995; Heraty & Morley, 1995; Mumford, 1993). Salaman 
(1995) has argued that key managerial competencies are “those that support the management of performance or the 
management of learning” (p. 5). However, there has been limited research into what managers do as coaches and 
facilitators of learning (Mumford, 1993; Heraty and Morley, 1995; Horowitz, 1999; IPD, 1995). Yet the increased 
attention to concepts such as workplace learning and the learning organization inevitably emphasizes the important 
role of managers as developers of others and the need to consider the developmental responsibilities and behaviors 
of line managers both from academic (e.g. Ellinger & Bostrom, 1999; Marsick & Watkins, 1999; Mumford, 1993) 
and policy perspectives (e.g. NSTF, 2000; IPD, 2000; SCVO, 1999; Scottish Executive, 2003; DoH, 2000, 
2002). and how such roles and behaviors can contribute to the development of learning organizations (Ellinger, 
1997). 

Current HRD practice has been criticised for lacking a sound and sufficient empirical base (Hamlin & Stewart, 
1998; Hamlin, 2002). Similar criticisms have been made of current coaching practice (Grant, 2003) and about 
devolution of HRD responsibilities to line managers (de Jong et ah, 1999). Whilst there is an extensive literature on 
developmental roles that managers may play, such as mentor and coach, much of this literature is prescriptive and 
there are few examples of substantive research and empirical studies (Beattie, 2002; Ellinger, 2003; Graham et ah, 
1993, 1994; Marsh, 1992; Popper and Lipshitz, 1992). To address this significant knowledge gap the three authors 
have conducted independent empirical studies, which were not influenced by each other’s work, to explore 
managerial behaviors in the Scottish non-profit/voluntary sector (Beattie, 2002), the English education, civil service 
and health sectors (Hamlin, 1990, 2002, 2004; Hamlin, Reidy & Stewart, 1998) and the US corporate sector 
(Ellinger, 1997). Following completion of these studies the authors have conducted a comparative analysis of their 
respective finding in search of commonalities and generic behaviors, and their findings have been reported 
accordingly (see Hamlin, Beattie and Ellinger, 2004; Beattie, Ellinger and Hamlin, 2004). 

However, a question arises regarding the efficacy of the methodological pluralism adopted for this comparative 
bearing in mind the different starting points and foci of their respective previous empirical studies, the different 
populations of managers studied, the different types of organizational settings chosen, and the variations in research 
methods deployed. Axelsson (1998) draws attention to the fact that most management research studies reported in 
the literature have been of the case study kind with few yielding results that can be generalized from one 
organization to another. In the field of leadership research Kim and Yukl (1995) report the number of studies of 
specific leadership and managerial leader behaviors is still small, but also different researchers have examined 
different sub-sets of behavior which in their view make it difficult to compare and contrasts the findings from one 
study with another. Furthermore, as Avolio, Bass and Jung (1999) argue, a major weakness in research design that 
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limits the generalizability of findings between various studies is the deployment of different people to gather data 
samples where there has been no central control over the consistency of the procedures utilized. To overcome such 
limitations they suggest researchers should conduct replica studies using common research designs and methods, in 
other words adopt a methodological unitary approach. 

Hence the focus of the present study has been to examine the perceived efficacy of the actual pluralistic 
approach deployed by the authors of this paper in their comparison of findings from their respective independent 
studies, with the aim of identifying the methodological challenges and benefits of this type of analytic comparative 
research. The paper begins by outlining the research questions that guided the three previous sets of studies, our 
recent collaborative study, and the specific research question guiding the present study. It then explores the 
methodological strategies that we each adopted and examines the similarities and differences in the approaches to 
data collection and analysis to highlight the extent of the methodological pluralism existing in this comparative 
study. The challenges and benefits of comparative research are then discussed, and the paper concludes that such an 
approach strengthens the validity and reliability of each independent study whilst collectively contributing 
significantly to the empirical literature base on managerial coaching and effectiveness which has been recognized as 
a critical area of HRD research and practice. 

Research Questions 

This section provides details of the research questions comprising the independent empirical studies of the three 
authors that provided the basis of the joint comparative study followed by the specific methodological research 
question driving the present study as follows. 

The Three Previous Empirical Studies 

Each author recognized that one aspect significantly lacking from the existing literature was a discussion and 
analysis of significant episodes or critical incidents from individuals’ job performance and/or careers where 
managers have had a positive (or negative) impact on the development of their employees. This view has been 
supported by Van der Krogt and Vermulst, 2000) who argue that “...the shift from training to learning [means] that 
managers and workers are now seen to have a considerable impact on the processes and structures of learning 
systems, alongside experts such as trainers and educational advisers. Accordingly, their views and insights with 
respect to their own actions and strategies have gained a more central position” (Van der Krogt & Vermulst, 2000). 
Each author attempted to gain such insights through the development of their research questions and methodological 
strategies. The research questions posed in each previously conducted studies are presented in Table 1, and 
demonstrate that the researchers were attempting to gather evidence about what managers actually do to help their 
employees learn and develop. 

Table 1. Research Questions 





Research Questions Guiding Each Study and the Collective Empirical Examination of the Studies 


Beattie (2002) 


1 . What do line managers do to facilitate learning? 

2. What do line managers do to inhibit learning? 

3. What motivates line managers to develop staff? 

4. What influence do individual factors have on developmental behaviors? 

5. What influence do organizational factors have on developmental behaviors? 


Ellinger 

(1997) 


1 . What are the managers’ beliefs about their role as facilitators of learning? 

2. What triggers the managers to engage in a learning episode? 

3. What are the role dimensions and related behaviors that contribute to the role of managers as 
facilitators of learning? 

4. What are the consequences for managers and learners associated with the learning episode? 


Hamlin 
(1990, 2002, 
2004); 

Hamlin et al. 
(1998) 


1. What specific managerial behaviors exhibited by middle and first line managers are 
perceived and judged to be concrete examples of either effective or ineffective management? 

2. What are the criteria of managerial and leadership effectiveness within the respective case 
study organization? 

3. To what extent are these criteria generalized within and to other organizations? 


Hamlin, 
Ellinger & 
Beattie (2004) 


1 . How are effective coaching /facilitating behaviors manifested within the studies of Hamlin 
(2004), Beattie (2002) and Ellinger (1997)? 

2. What behaviors are held in common and generalized to each other? 
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Present study 

The specific methodological research question selected for the present study was as follows: To what extent can 
methodological pluralism and analytical triangulation be meaningfully utilized for identifying commonalities and 
generalized findings between the three sets of managerial behaviors obtained from three previous empirical studies 
conducted wholly independently of each other? 

Research Strategies 

The methodological strategies adopted by each researcher were remarkably similar in terms of research philosophy 
and research design. Differences emerged in the approach to data analysis. Each research design is reported briefly 
below. See Table 2 for an overview of the research strategies. 

Beattie (2002) adopted a predominantly phenomenological approach, utilizing intensive field studies. A study 
of actors to create a shared meaning system or world view was regarded as the only way to produce the type of 
knowledge desired (Andersen et ak, 1995). Due to lack of insight into processes or the significance that people 
apply to actions, quantitative methods were considered inappropriate for research trying to understand behavior in 
specific situations and how people felt about their developmental interactions. However, her considered view was 
that using qualitative methods, including participant observation and interviews, within overarching descriptive and 
explanatory case studies (Yin, 1989) could facilitate: the understanding of how people feel about developmental 
relationships; the development of new ideas about developmental relationships; and ultimately, contribute to new 
theory thus enhancing our understanding of management and workplace learning. A further reason for utilizing a 
phenomenological approach was that the sector studied, the non-profit/voluntary sector, is under-researched in the 
UK in comparison to the public and corporate sectors. 

Ellinger (1997) also adopted a qualitative research design that employed in-depth semi-structured interviews 
and the critical incident technique as the primary approaches to data collection for her study in the U.S. corporate 
sector. Her rationale was that qualitative methods, unlike quantitative methods, allow the researcher to study 
selected issues in depth and detail (Patton, 1990). Qualitative methods are also considered to be most appropriate 
when there is little knowledge about a phenomenon (Merriam & Simpson, 1995), as is the case with line managers 
as facilitators of learning. Although generalizability is reduced, issues related to validity depend on the construction 
of a reliable research protocol and the administration of the protocol in a standardized manner. In qualitative 
research, “the researcher is the instrument” (Patton, 1990, p.l4), and therefore, validity depends largely on the skill, 
competence and rigor of the researcher doing the fieldwork. Hamlin (1990, 2002) and Hamlin, Reidy and Stewart 
(1998), who have carried out three empirical studies into managerial effectiveness in different organizations across 
the UK public sector, likewise, adopted a qualitative approach for their respective initial data collections. However, 
they borrowed from the positivistic tradition by also developing a survey instrument; this is discussed more fully 
below. 

Table 2. Overview of Research Strategies 





Beattie (2002) 


Ellinger (1997) 


Hamlin (2004) 


Sample Organizations 


2 voluntary sector 
organizations 


4 corporate learning 
organizations 


3 public sector organizations: 

• High Schools 

• Civil Service Department 

• NHS Trust Hospital 


Data Collection 
Methods 


• Participant 
observation 

• Semi-structured 
interviews (CIT) 

• Documentary 
analysis 


• Semi-structured 
interviews 

• (CIT) 


• Semi-structured interviews 
(CIT) 

• Summarising, explicative and 
structuring content analysis to 
create Behavioral Item 
Questionnaires (BIQs) 


Research Population 


• Senior Line 
Managers (n=13) 

• Eirst Line 
Managers (n=12) 

• Employees (n=35) 


• Managers (n= 12) 


• CIT Interviews: 

• Managers (n=222) 

• Administering the BIQ 

• Top-down perspective (n=328) 

• Self-perspective (n=148) 
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• Bottom-up perspective (n=44) 


Data Analysis 


• Grounded theory 

• Manager profile 
analysis 


• Content analysis 

• Manager profile 
analysis 

• Open coding 


• Factor analysis and principal 
component analysis 

• Content analysis 

• Open coding 



Discussion 

Data Collection 

The two organizations selected for Beattie’s study were two major Scottish voluntary sector social care 
providers who were regarded as ‘employers-of-choice’ in their respective fields, were recognized as ‘Investors in 
People’ and have aspirations to become learning organizations. Beattie’s study included senior line managers, line 
managers, and their respective employees. The organizations selected for Ellinger’s study were four private sector 
organizations highlighted by Watkins and Marsick’s (1996) study of learning organizations. The managers 
participating in Ellinger’s study included senior and mid-level managers representing diverse functional areas within 
their respective organizations. Similarly, unlike most management and leadership research carried out since the 
1980s which has been concerned predominantly with studying the behaviors of top managers and organizational 
leaders, Hamlin’s studies focused instead on middle and first line managers and these studies were conducted within 
public sector organizations representing education, healthcare, and government. 

Commonalities . Each author used in-depth semi-structured interviews as the principal research instrument. 
Easterby-Smith at al (1991) argue that such interviews are appropriate when: It is necessary to understand the 
constructs the interviewee uses as a basis for opinions and beliefs about a particular matter or situation; 
Furthermore, there is a need to understand the respondent’s ‘world’ so that the researcher may influence it (e.g. 
through action research); and that step-by-step logic of a situation is unclear; subject matter is highly confidential or 
commercially sensitive; and, an interviewee may only be truthful in the confidentiality of a one-to-one situation. 

Such interviews were therefore appropriate for these studies into managerial behaviors as there was a need to 
understand respondents’ opinions about developmental incidents and relationships, and prior to these studies there 
had been little understanding of such relationships due to the limited research to date. Although there was limited 
commercial sensitivity for this topic there was potentially a high degree of personal sensitivity, and therefore 
respondents may have been more comfortable and open in a one-to-one situation. There was no intention to 
influence the respondents’ worlds through the likes of action research or co-operative inquiry. However, the authors 
hope that these individual and comparatives studies would influence the development and behaviors of managers in 
the long-term. 

Also common to each author’s work was the use of Critical Incident Technique (CIT) within the research 
interview setting. This is an approach which “...rather than collecting opinions, hunches and estimates, obtains a 
record of specific behaviors from those in the best position to make the necessary observations and evaluations 
(Flanagan, 1954, p.355). The CIT is a systematic and sequential method for collecting observed incidents, or 
observations previously made which are reported from memory. As a method for classification, it can be 
categorized with other inductive grouping procedures such as factor analysis, cluster analysis and multidimensional 
scaling (Bitner et al., 1990). Within such grouping procedures, the researcher ‘lets the data suggest the system’ 
(Hunt, 1991, p. 178), and as such, a classification or schema is generated only after the data area has been analyzed. 
The researcher does not impose a classification system on the data. Unlike the other grouping procedures, the CIT 
typically uses content analysis, and as in Beattie’s study, grounded theory, rather than quantitative solutions in the 
data analysis. Table 3 below synthesizes the benefits of the CIT. 

This method has been used by qualitative researchers to great effect with in-depth interviews (Easterby-Smith et 
al., 1991), and particularly to research natural processes of learning at work (Easterby-Smith & Thorpe, 1997). 
However, the technique has been criticised because of the potential difficulty of recall and the ability of individuals 
to focus on actual behaviors. By asking them to track back to particular instances in their work lives and to explain 
their actions and motives with specific regard to those instances (ibid. p. 83), and by limiting permitted recall to 
critical incidents that have occurred in recent memory, namely within the preceding 6 to 9 months, helps to 
minimise the limitations of hindsight. Indeed Marsick and Watkins (1999) argue that there is a need for more 
studies, such as these, which focus on the reflective learning/intervention relationship. This technique also supports 
the application of interpretative interactionism, which focuses on epiphanies: ...those interactional moments that 
leave marks on people ’s lives [and] have the potential for creating transformational experiences for the person 
(Denzin, 1989, p.l5). Critical incidents in the respective studies were defined as being learning episodes between a 



28-2 





652 



manager and his/her employees where the manager/employee believed the manager had facilitated or inhibited the 
employee(s)’ learning. 

Table 3: Benefits of Critical Incident Technique (Ellinger, 1997 adapted from Clawson, 1992) 

Benefits of Critical Incident Technique 

• Emphasis on observable behaviors or behaviors recalled from memory 

• Collected from ‘actor’s’ perspective vs. the opinions of outside experts 

• Reinforces idea of ‘the answers are in the model’ 

• Especially useful in determining detailed situational behavior 

• Develops a comprehensive picture 

• Best technique for developing behavioral criteria 

• Ability to discover both universal and contingent behavior 

• Proven methodology - used with considerable documented success 

• Generates rich qualitative data 



Distinctions. Each study utilized the CIT as an approach to data collection. However, Hamlin’s study also used 
the results of the CIT to generate behavioral item questionnaires, which then became a primary tool for data 
collection and analysis. Beattie also adopted the use of participant observation for her study to gain an 
understanding of the language and culture of her respondents prior to conducting interviews (Eontana and Erey, 
1994). While the overall samples in each study included line managers and mid-level managers, Beattie’s and 
Hamlin’s studies incorporated managers’ respective employees and represent an area of distinction from Ellinger’ s 
work. 

Data Analysis 

It was at their respective data analysis stages that we see the main differences in methodological approach. 
Whilst Ellinger (1997) and Beattie (2002) developed managerial profiles utilizing data gathered on individual 
managers, the process they used to develop these profiles was significantly different. Beattie (2002) utilized a 
grounded theory approach to analyzing her data. The rationale for this was that grounded theory enabled the 
researcher to develop theory through ‘comparative method’ by conducting in-depth interviews with different 
developmental managers in different voluntary organizations. Andersen et al (1995) argue that if we accept that 
scientific theories are merely capable of capturing limited aspects of the phenomenon being studied, the theories of 
the field - where the social field’s creation of meaning and use of common-sense concepts enables the inhabitants to 
manage everyday life - must be granted a similar status to that of the academic field. This view of the social field’s 
development falls within the grounded theory tradition and therefore theory and empirical work should interact. 
Building on this philosophical approach Beattie utilized Easterby-Smith et al’s (1991) seven stage framework for 
analyzing in-depth interview data including: familiarization, reflection, conceptualization, cataloguing concepts, 
recoding, linking and re-evaluation. The findings that emerged from this analysis were then tested for validity with 
members of the research population. 

The primary approach to data analysis in Ellinger’ s (1997) study was content analysis. Content analysis uses a 
set of procedures to make valid inferences from text, which include inferences about the sender(s) of the message, 
the message, or the audience of the message (Weber, 1990). The use of content analysis aids in the classification of 
textual material thereby reducing it to more relevant and manageable bits of data. Ellinger drew upon four broad 
categories of the conceptual framework guiding her study as an organizing framework to begin sorting the data since 
content analysis requires the design and implementation of coding schemes. In this phase of content analysis, the 
researcher creates the category of definitions and determines if the categories are to be mutually exclusive. Once 
content analysis was completed, the next phase of data analysis involved open coding. The open coding process was 
used to sort the data within the framework categories. 

Hamlin (2002, 2004) utilized a quantitative approach in his data analysis, namely principal component analysis 
and factor analysis to reduce and classify the data and to identify behavioral categories/criteria of effectiveness. Eor 
each study ‘summarizing’, ‘explicative’ and ‘structuring’ content analysis was then used to identify from the 
gathered CIT data discrete behavioral statements (items). These were then used to develop a Behavioral Item 
Questionnaire (BIQ) (see Latham & Wexley, 1981). The respective BIQs were subsequently administered to 
managers and non-managers within the case study organizations, namely the middle and first line managers 
themselves who were the focus of the studies (self-rating), their superiors (top down rating) and their subordinates 
(bottom up rating). The data so obtained was then factor analyzed to identify the criteria of managerial effectiveness 
applying within the respective organizations. In subsequent studies, Hamlin (2003, 2004) carried out meta-level 
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analyzes in search of evidence of the external generalizability of these previously derived criteria using a variant of 
open coding applied inductively and deductively within a grounded theory mindset. 

Conclusions and contributions to HRD 

The key challenge is this comparative study was the limitations associated with the methods used and sampling. 
Although the CIT was used according to the protocols outlined by Flanagan (1954) in all three studies, it is possible 
that the richness and depth of the critical incident data collected may have varied within each of the three sets of 
independent studies. It is also possible that data from interviews, observations, and documents also assisted in the 
analyses and informed each of the three respective studies. Another limitation in conducting this meta-level 
comparison has been the fact that the original samples of managers for each of the three studies also varied to some 
degree. The Hamlin study (2004) was generated based upon the collection of critical incident data from large and 
homogeneous groups of middle and first line managers within six public organizations with the intent to examine 
manager and leader behavior in general. In slight contrast, Ellinger (1997) and Ellinger and Bostrom’s (1999) 
sample was comprised of mid-level and senior level exemplars who were nominated as being exemplary learning 
facilitators within four consumer goods and manufacturing organizations deemed to be learning organizations. 
Similarly, Beattie’s (2002) sample was comprised of line managers, senior line managers and employees in 
voluntary sector social service organizations, which had aspirations to become learning organizations. Both the 
Ellinger (1997) and Beattie (2002) studies were designed to specifically examine coaching/facilitating learning 
behaviors. Despite these limitations, it appears, following bi-lateral and tri-lateral analyses, that the findings across 
all three studies are robust and the meta-level comparisons provide considerable evidence about critical managerial 
coaching behaviors. 



Table 4. Summary of the Major Bindings from Each Author’s Study 



Beattie’s Nine Behavioral Categories: 


•Caring 


•Thinking 


•Informing 


•Empowering 


•Being Professional 


•Developing Developers 


•Advising 


•Challenging 


•Assessing 




Ellinger’ s Thirteen Behaviors: 


•Question framing to encourage employees to think 


•Providing feedback to employees 


through issues 




•Being a resource - removing obstacles 


•Soliciting feedback from employees 


•Transferring ownership to employees 


•Working it out together - talking it through 


•Holding back - not providing the answers 


•Creating and promoting a learning environment 




•Setting and communicating expectations - fitting into 




the big picture 




•Stepping into other to shift perspectives 




•Broadening employees’ perspectives - getting them to 




see things differently 




•Using analogies, scenarios, and examples 




•Engaging others to facilitate learning 


Hamlin’s Six Positive Behaviors 


•Effective organization and planning/proactive 


•Genuine concern for people/Looks after the interests 


management 


and development needs of staff 


•Participative and supportive leadership/proactive team 


•Open and personal management approach/inclusive 


leadership 


decision making 


•Empowerment and delegation 


•Communicates, informs and consults widely 



As Table 4 above illustrates, despite these variations in data analysis approaches, considerable similarities were 
found in the behavioral findings that emerged within each of the three studies. Although the focus of this paper 
relates to the design and analysis approaches that each study employed, considerable overlap among the findings of 
all three studies has been found in our other collaborative work (see e.g. Hamlin, Ellinger & Beattie, 2004; Beattie, 
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Ellinger & Hamlin, 2004). Part of the reason for this comparative study across three qualitative researchers’ work 
has also been to test the rigor, trustworthiness of each author’s work, and to explore for evidence of generalizability. 

This paper therefore demonstrates despite some variations in aspects of the data collection and analyses phases 
of our independent work, we have methodologically illustrated triangulation among independent studies with the 
purpose of theory-building. Through this comparative analysis of primary data we have gone beyond merely 
reviewing our respective literature reviews and findings, and have attempted to integrate and synthesize the findings 
to add more richly to this important field of HRD. This comparative study has also addressed concerns expressed in 
the research methods literature about the lack of generalizability of qualitative case study research. We have also 
addressed a concern of the Academy of Management (2001) about the lack of interaction between qualitative and 
quantitative research methods through our synergistic use of methods, particularly in terms of data analysis. 

Finally, we would argue that work such as this both builds the evidence-base for HRD policy and practice, and 
demonstrates the value of international collaborative research, particularly when the collaborators come from 
cultures with different research and discipline traditions. Indeed it is our intention to continue such work and further 
develop theory in this field through replica studies. 
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